Networks For Human Activity Recognition Research Articles

ActiWise: Insight on Human Activity Recognition Using Deep Learning Approaches

In this study, we investigate the fusion of Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks for human activity recognition (HAR). By integrating hierarchical spatial features extracted by CNNs with LSTM networks' temporal modelling capabilities, our approach excels in discerning nuanced patterns from raw sensor data collected via wearable devices. Through rigorous experimentation and validation, our CNN+LSTM model demonstrates robust performance in accurately classifying a spectrum of human activities. This research advances HAR methodologies, shedding light on the synergistic interplay between spatial and temporal modelling in activity recognition, with implications across healthcare, sports analytics, and human-computer interaction domains. Index Terms- Human activity recognition, deep learning, Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM).

Various frameworks for integrating image and video streams for spatiotemporal information learning employing 2D–3D residual networks for human action recognition

Human action recognition has been identified as an important research topic in computer vision because it is an essential form of communication and interplay between computers and humans to assist computers in automatically recognizing human behaviors and accurately comprehending human intentions. Inspired by some keyframe extraction and multifeatured fusion research, this paper improved the accuracy of action recognition by utilizing keyframe features and fusing them with video features. In this article, we suggest a novel multi-stream approach architecture made up of two distinct models fused using different fusion techniques. The first model combines convolutional neural networks in two-dimensional (2D-CNN) with long-short term memory networks to glean long-term spatial and temporal features from video keyframe images for human action recognition. The second model is a three-dimensional convolutional neural network (3D-CNN) that gathers quick spatial–temporal features from video clips. Subsequently, two frameworks are put forth to explain how various fusion structures can improve the performance of action recognition. We investigate methods for video action recognition using early and late fusion. While the late-fusion framework addresses the decision fusion from the two models' choices for action recognition, the early-fusion framework examines the impact of early feature fusion of the two models for action recognition. The various fusion techniques investigate how much each spatial and temporal feature influences the recognition model's accuracy. The HMDB-51 and UCF-101 datasets are two important action recognition benchmarks used to evaluate our method. When applied to the HMDB-51 dataset and the UCF-101 dataset, the early-fusion strategy achieves an accuracy of 70.1 and 95.5%, respectively, while the late-fusion strategy achieves an accuracy of 77.7 and 97.5%, respectively.

Leveraging spatio-temporal features using graph neural networks for human activity recognition

Unsupervised human activity recognition (HAR) algorithms working on motion capture (mocap) data often use spatial information and neglect the activity-specific information contained in the temporal sequences. In this work, we propose a new unsupervised algorithm for HAR from mocap data to leverage both spatial and temporal information embedded in activity sequences. For this, we employ a shallow graph neural network (GNN) comprising a graph convolutional network and a gated recurrent unit to aggregate the spatial and temporal features of the mocap sequences, respectively. Moreover, we encode the transformations of the human body through log-regularized kernel covariance descriptors linked to the trajectory movement maps of mocap frames. These descriptors are then fused with the GNN features for downstream activity recognition tasks. Finally, HAR is performed by a new unsupervised algorithm using a neighborhood Laplacian regularizer and a normalized dictionary learning approach. The generalizability of the proposed model is validated by training the GNN on a public dataset and testing on the other datasets. The performance of the proposed model is evaluated using six publicly available human mocap datasets. Compared to existing approaches, the proposed model improves activity recognition consistently by 12%–30% across different datasets.

Wearable Sensor-Based Residual Multifeature Fusion Shrinkage Networks for Human Activity Recognition.

Human activity recognition (HAR) based on wearable sensors has emerged as a low-cost key-enabling technology for applications such as human-computer interaction and healthcare. In wearable sensor-based HAR, deep learning is desired for extracting human active features. Due to the spatiotemporal dynamic of human activity, a special deep learning network for recognizing the temporal continuous activities of humans is required to improve the recognition accuracy for supporting advanced HAR applications. To this end, a residual multifeature fusion shrinkage network (RMFSN) is proposed. The RMFSN is an improved residual network which consists of a multi-branch framework, a channel attention shrinkage block (CASB), and a classifier network. The special multi-branch framework utilizes a 1D-CNN, a lightweight temporal attention mechanism, and a multi-scale feature extraction method to capture diverse activity features via multiple branches. The CASB is proposed to automatically select key features from the diverse features for each activity, and the classifier network outputs the final recognition results. Experimental results have shown that the accuracy of the proposed RMFSN for the public datasets UCI-HAR, WISDM, and OPPORTUNITY are 98.13%, 98.35%, and 93.89%, respectively. In comparison with existing advanced methods, the proposed RMFSN could achieve higher accuracy while requiring fewer model parameters.

Channel Attention-Based Approach with Autoencoder Network for Human Action Recognition in Low-Resolution Frames

Action recognition (AR) has many applications, including surveillance, health/disabilities care, man-machine interactions, video-content-based monitoring, and activity recognition. Because human action videos contain a large number of frames, implemented models must minimize computation by reducing the number, size, and resolution of frames. We propose an improved method for detecting human actions in low-size and low-resolution videos by employing convolutional neural networks (CNNs) with channel attention mechanisms (CAMs) and autoencoders (AEs). By enhancing blocks with more representative features, convolutional layers extract discriminating features from various networks. Additionally, we use random sampling of frames before main processing to improve accuracy while employing less data. The goal is to increase performance while overcoming challenges such as overfitting, computational complexity, and uncertainty by utilizing CNN-CAM and AE. Identifying patterns and features associated with selective high-level performance is the next step. To validate the method, low-resolution and low-size video frames were used in the UCF50, UCF101, and HMDB51 datasets. Additionally, the algorithm has relatively minimal computational complexity. Consequently, the proposed method performs satisfactorily compared to other similar methods. It has accuracy estimates of 77.29, 98.87, and 97.16%, respectively, for HMDB51, UCF50, and UCF101 datasets. These results indicate that the method can effectively classify human actions. Furthermore, the proposed method can be used as a processing model for low-resolution and low-size video frames.

Multi-STMT: Multi-Level Network for Human Activity Recognition Based on Wearable Sensors

Multi-STMT: Multi-Level Network for Human Activity Recognition Based on Wearable Sensors

A Study on the Energy Sustainability of Early Exit Networks for Human Activity Recognition

A Study on the Energy Sustainability of Early Exit Networks for Human Activity Recognition

A Dense-Sparse Complementary Network for Human Action Recognition based on RGB and Skeleton Modalities

The vulnerability of RGB-based human action recognition in complex environment and variational scenes can be compensated by skeleton modality. Therefore, action recognition methods fusing RGB and skeleton modalities have received increasing attention. However, the recognition performance of the existing methods is still not satisfactory due to the insufficiently optimized sampling, modeling and fusion strategy, even the computational cost is heavy. In this paper, we propose a Dense-Sparse Complementary Network (DSCNet), which aims to leverage the complementary information of the RGB and skeleton modalities at light computational cost to obtain the competitive action recognition performance. Specifically, we first adopt dense and sparse sampling strategies according to the advantages of RGB and skeleton modalities, respectively. And then, we use the skeleton as guiding information to crop the key active region of the persons in the RGB frame, which largely eliminates the interference of the background. Moreover, a Short-Term Motion Extraction Module (STMEM) is proposed to compress the densely sampled RGB frames to fewer frames before feeding them into the backbone network, which avoids a surge in computational cost. And a Sparse Multi-Scale Spatial–Temporal convolutional neural Network (Sparse-MSSTNet) is designed to modeling sparse skeleton. Extensive experiments show that our method effectively combines complementary information of RGB and skeleton modalities to improve recognition accuracy. The DSCNet achieves competitive performance on NTU RGB+D 60, NTU RGB+D 120, PKU-MMD, UAV-human, IKEA ASM and Northwest-UCLA datasets with much less computational cost than exiting methods. The code is available at https://github.com/Maxchengqin/DSCNet.

Residual Neural Networks for Human Action Recognition from RGB-D Videos

Recently, the RGB-D based Human Action Recognition (HAR) has gained significant research attention due to the provision of complimentary information by different data modalities. However, the current models have experienced still unsatisfactory results due to several problems including noises and view point variations between different actions. To sort out these problems, this paper proposes two new action descriptors namely Modified Depth Motion Map (MDMM) and Spherical Redundant Joint Descriptor (SRJD). MDMM eliminates the noises from depth maps and preserves only the action related information. Further SRJD ensures resilience against view point variations and reduces the misclassifications between different actions with similar view properties. Further, to maximize the recognition accuracy, standard deep learning algorithm called as Residual Neural Network (ResNet) is used to train the system through the features extracted from MDMM and SRJD. Simulation experiments prove that the multiple data modalities are better than single data modality. The proposed approach was tested on two public datasets namely NTURGB+D dataset and UTD-MHAD dataset. The testing results declare that the proposed approach is superior to the earlier HAR methods. On an average, the proposed system gained an accuracy of 90.0442% and 92.3850% at Cross-subject and Cross-view validations respectively.

AReNet: Cascade learning of multibranch convolutional neural networks for human activity recognition

AReNet: Cascade learning of multibranch convolutional neural networks for human activity recognition

MAG-Res2Net: a novel deep learning network for human activity recognition

Objective. Human activity recognition (HAR) has become increasingly important in healthcare, sports, and fitness domains due to its wide range of applications. However, existing deep learning based HAR methods often overlook the challenges posed by the diversity of human activities and data quality, which can make feature extraction difficult. To address these issues, we propose a new neural network model called MAG-Res2Net, which incorporates the Borderline-SMOTE data upsampling algorithm, a loss function combination algorithm based on metric learning, and the Lion optimization algorithm. Approach. We evaluated the proposed method on two commonly utilized public datasets, UCI-HAR and WISDM, and leveraged the CSL-SHARE multimodal human activity recognition dataset for comparison with state-of-the-art models. Main results. On the UCI-HAR dataset, our model achieved accuracy, F1-macro, and F1-weighted scores of 94.44%, 94.38%, and 94.26%, respectively. On the WISDM dataset, the corresponding scores were 98.32%, 97.26%, and 98.42%, respectively. Significance. The proposed MAG-Res2Net model demonstrates robust multimodal performance, with each module successfully enhancing model capabilities. Additionally, our model surpasses current human activity recognition neural networks on both evaluation metrics and training efficiency. Source code of this work is available at: https://github.com/LHY1007/MAG-Res2Net.

Multi-model weighted voting method based on convolutional neural network for human activity recognition

Multi-model weighted voting method based on convolutional neural network for human activity recognition

Extension of physical activity recognition with 3D CNN using encrypted multiple sensory data to federated learning based on multi-key homomorphic encryption

Background and Objective: The Internet of medical things is enhancing smart healthcare services using physical wearable sensor-based devices connected to the Internet. Machine learning techniques play an important role in the core of these services for remotely consulting patients thanks to the pattern recognition from on-device data, which is transferred to the central servers from local devices. However, transferring personally identifiable information data to servers could become a source for hackers to steal from, manipulate and perform illegal activities. Federated learning is a new branch of machine learning that creates directly training models from on-device data and aggregates these learned models on the servers without centralized data. Another way to protect data confidentiality on computer systems is data encryption. Data encryption transforms data into another form that only users with authority to a decryption key can read. In this work, we propose a novel method enabling preservation of client privacy and protection of client biomedical data from illegal hackers while transmitting through the Internet.Methods: We propose a method applying 3-dimensional convolutional neural networks for human activity recognition using multiple sensory data. In order to protect the data, we apply the bitwise XOR operator encryption technique. Then, we extend our 3-dimensional convolutional neural network methods to both traditional federated learning and the federated learning based on multi-key homomorphic encryption using the proposed encrypting data.Results: Based on leave-one-out-cross-validation, the 3-dimensional method obtains an accuracy of 94.6% and of 94.9% (without data encrypting and without federated learning) tested on two different benchmarked datasets, Sport and DaLiAC respectively. Accuracy is decreased slightly to 89.5% (from 94.6% of the baseline) when we use the proposed encrypting data method. However, the encryption-data-based method still has a potential result compared to the state-of-the-art which only uses raw data. In addition, the proposed full federated learning scheme of this work shows that illegal persons who somehow can get the trained model transmitted via networks cannot infer the private result.Conclusions: This novel method for sensory data representation which translates temporal and frequency bio-signal values to voxel intensities that can encode 3-dimensional activity images. Secondly, the proposed 3-dimensional convolutional neural network methods outperform other deep-learning-based human activity recognition approaches. Finally, extensive experiments show the proposed data-encrypted federated learning approach can achieve feasibility in terms of efficiency in privacy preservation.

A Low-Cost Wireless Body Area Network for Human Activity Recognition in Healthy Life and Medical Applications

A Low-Cost Wireless Body Area Network for Human Activity Recognition in Healthy Life and Medical Applications

A heterogeneous two-stream network for human action recognition

The most widely used two-stream architectures and building blocks for human action recognition in videos generally consist of 2D or 3D convolution neural networks. 3D convolution can abstract motion messages between video frames, which is essential for video classification. 3D convolution neural networks usually obtain good performance compared with 2D cases, however it also increases computational cost. In this paper, we propose a heterogeneous two-stream architecture which incorporates two convolutional networks. One uses a mixed convolution network (MCN), which combines some 3D convolutions in the middle of 2D convolutions to train RGB frames, another one adopts BN-Inception network to train Optical Flow frames. Considering the redundancy of neighborhood video frames, we adopt a sparse sampling strategy to decrease the computational cost. Our architecture is trained and evaluated on the standard video actions benchmarks of HMDB51 and UCF101. Experimental results show our approach obtains the state-of-the-art performance on the datasets of HMDB51 (73.04%) and UCF101 (95.27%).

Attention-Based Hybrid Deep Learning Network for Human Activity Recognition Using WiFi Channel State Information

The recognition of human movements is a crucial aspect of AI-related research fields. Although methods using vision and sensors provide more valuable data, they come at the expense of inconvenience to users and social limitations including privacy issues. WiFi-based sensing methods are increasingly being used to collect data on human activity due to their ubiquity, versatility, and high performance. Channel state information (CSI), a characteristic of WiFi signals, can be employed to identify various human activities. Traditional machine learning approaches depend on manually designed features, so recent studies propose leveraging deep learning capabilities to automatically extract features from raw CSI data. This research introduces a versatile framework for recognizing human activities by utilizing CSI data and evaluates its effectiveness on different deep learning networks. A hybrid deep learning network called CNN-GRU-AttNet is proposed to automatically extract informative spatial-temporal features from raw CSI data and efficiently classify activities. The effectiveness of a hybrid model is assessed by comparing it with five conventional deep learning models (CNN, LSTM, BiLSTM, GRU, and BiGRU) on two widely recognized benchmark datasets (CSI-HAR and StanWiFi). The experimental results demonstrate that the CNN-GRU-AttNet model surpasses previous state-of-the-art techniques, leading to an average accuracy improvement of up to 4.62%. Therefore, the proposed hybrid model is suitable for identifying human actions using CSI data.

Temporal-channel convolution with self-attention network for human activity recognition using wearable sensors

Human activity recognition (HAR) is an essential task in many applications such as health monitoring, rehabilitation, and sports training. Sensor-based HAR has received increasing attention due to the widespread availability of sensors in daily life. In this paper, we propose two novel architectures, the convolution with self-attention network (CSNet) and the temporal-channel convolution with self-attention network (TCCSNet), for classifying sequences of human activity data from different sensors. CSNet leverages both convolution and self-attention to capture both local and global dependencies in the input data, while TCCSNet exploits both temporal and inter-channel dependencies through two branches of convolutions and self-attentions for extracting time-wise and channel-wise information. The proposed methods are evaluated on seven different sensor-based HAR datasets, namely: MHEALTH, PAMAP2, UTD1, UTD2, WHARF, USC-HAD, and WISDM, using the leave-one-subject-out cross-validation protocol. Our experiments show that the proposed models outperform other modern approaches, such as Transformers and long short-term memory (LSTM) based models.

A Prior Knowledge-Guided Graph Convolutional Neural Network for Human Action Recognition in Solar Panel Installation Process

Human action recognition algorithms have garnered significant research interest due to their vast potential for applications. Existing human behavior recognition algorithms primarily focus on recognizing general behaviors using a large number of datasets. However, in industrial applications, there are typically constraints such as limited sample sizes and high accuracy requirements, necessitating algorithmic improvements. This article proposes a graph convolution neural network model that combines prior knowledge supervision and attention mechanisms, designed to fulfill the specific action recognition requirements for workers installing solar panels. The model extracts prior knowledge from training data, improving the training effectiveness of action recognition models and enhancing the recognition reliability of special actions. The experimental results demonstrate that the method proposed in this paper surpasses traditional models in terms of recognizing solar panel installation actions accurately. The proposed method satisfies the need for highly accurate recognition of designated person behavior in industrial applications, showing promising application prospects.

Fine-Tuned Temporal Dense Sampling with 1D Convolutional Neural Network for Human Action Recognition.

Human action recognition is a constantly evolving field that is driven by numerous applications. In recent years, significant progress has been made in this area due to the development of advanced representation learning techniques. Despite this progress, human action recognition still poses significant challenges, particularly due to the unpredictable variations in the visual appearance of an image sequence. To address these challenges, we propose the fine-tuned temporal dense sampling with 1D convolutional neural network (FTDS-1DConvNet). Our method involves the use of temporal segmentation and temporal dense sampling, which help to capture the most important features of a human action video. First, the human action video is partitioned into segments through temporal segmentation. Each segment is then processed through a fine-tuned Inception-ResNet-V2 model, where max pooling is performed along the temporal axis to encode the most significant features as a fixed-length representation. This representation is then fed into a 1DConvNet for further representation learning and classification. The experiments on UCF101 and HMDB51 demonstrate that the proposed FTDS-1DConvNet outperforms the state-of-the-art methods, with a classification accuracy of 88.43% on the UCF101 dataset and 56.23% on the HMDB51 dataset.

Design of cuckoo search optimization with deep belief network for human activity recognition and classification

Design of cuckoo search optimization with deep belief network for human activity recognition and classification